Sourced from datasets's releases.

4.0.0

New Features

• Add IterableDataset.push_to_hub() by @​lhoestq in huggingface/datasets#7595

  # Build streaming data pipelines in a few lines of code !
  from datasets import load_dataset
  ds = load_dataset(..., streaming=True)
  ds = ds.map(...).filter(...)
  ds.push_to_hub(...)

• Add num_proc= to .push_to_hub() (Dataset and IterableDataset) by @​lhoestq in huggingface/datasets#7606

  # Faster push to Hub ! Available for both Dataset and IterableDataset
  ds.push_to_hub(..., num_proc=8)

• New Column object

  • Implementation of iteration over values of a column in an IterableDataset object by @​TopCoder2K in huggingface/datasets#7564
  • Lazy column by @​lhoestq in huggingface/datasets#7614

  # Syntax:
  ds["column_name"]  # datasets.Column([...]) or datasets.IterableColumn(...)
  Iterate on a column:
  for text in ds["text"]:
  ...
  Load one cell without bringing the full column in memory
  first_text = ds["text"][0]  # equivalent to ds[0]["text"]

• Torchcodec decoding by @​TyTodd in huggingface/datasets#7616

  • Enables streaming only the ranges you need !

  # Don't download full audios/videos when it's not necessary
  # Now with torchcodec it only streams the required ranges/frames:
  from datasets import load_dataset
  ds = load_dataset(..., streaming=True)
  for example in ds:
  video = example["video"]
  frames = video.get_frames_in_range(start=0, stop=6, step=1)  # only stream certain frames

  • Requires torch>=2.7.0 and FFmpeg >= 4

... (truncated)

• b0de7a8 Release: 4.0.0 (#7673)
• c4bdfe8 Fix double sequence (#7672)
• 8a19de0 Fix audio bytes (#7670)
• c2490a4 Fix infer list of images (#7667)
• b5a2a94 Backward compat list feature (#7666)
• cc40530 Update the beans dataset link in Preprocess (#7659)
• 970a0df Custom metadata filenames (#7663)
• e71b0b1 fix sequence ci (#7644)
• 28b5e24 Backward compat sequence instance (#7643)
• cefda74 fix length for ci (#7642)
• Additional commits viewable in compare view

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge will merge this PR after your CI passes on it
• @dependabot squash and merge will squash and merge this PR after your CI passes on it
• @dependabot cancel merge will cancel a previously requested merge and block automerging
• @dependabot reopen will reopen this PR if it is closed
• @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)